Skip to content

Overlay: Add overlay annotations to Java & shared libraries #19779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: kaspersv/overlay-annotations-script
Choose a base branch
from

Conversation

kaspersv
Copy link
Contributor

@kaspersv kaspersv commented Jun 16, 2025

This PR adds overlay annotations for Java libraries and shared libraries to support experimentation with Java overlay analysis. Overlay annotations were added automatically using the add-overlay-annotations.py script. The high-level intend is that dataflow should be global and as much as possible below dataflow should be local. To achieve this the script adds top-level overlay[local?] annotations to Java and shared libraries based on a simple heuristic:

  • skip library files that end with Test.qll
  • skip library files that end with Query.qll or Config.qll if they contain implements DataFlow::ConfigSig.

For files selected for annotation, the script also adds overlay[caller] annotations on all public predicates annotated with pragma[inline] to ensure that those predicates will still be inlined across the overlay frontier once overlay compilation is enabled. See the internal Incremental CodeQL docs for additional details.

I recommend reviewing this PR by reviewing the script that generated the annotations.

Some of the annotated Java library files are also used by Python and C# and the annotations have therefore also been added to the Python and C# variants through sync-files.py.

This PR does not enable overlay compilation for Java and therefore currently has no effect on the generated DIL, RA or QLX (as witnessed by the uneventful DCA runs). The only current impact of adding the annotations will be that the compiler will check for overlay annotation errors, but in the absence of errors the overlay annotations currently have no effect on compilation. See the internal Incremental CodeQL docs for additional details.

A CI check will be added in a subsequent PR to enforce usage of the add-overlay-annotations.py script to automatically add overlay annotations to newly added files.

@kaspersv kaspersv marked this pull request as ready for review June 18, 2025 06:56
@Copilot Copilot AI review requested due to automatic review settings June 18, 2025 06:56
@kaspersv kaspersv requested review from a team as code owners June 18, 2025 06:56
@kaspersv kaspersv added the no-change-note-required This PR does not need a change note label Jun 18, 2025
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds overlay[local?] and module; declarations across Java (and some C#) QL library files to opt them into the new overlay compilation mode, and applies overlay[caller] to inline public predicates to preserve inlining across overlay boundaries.

  • Insert overlay[local?] and module; at the top of selected Java/QSharp QLL files following the heuristic in the PR description.
  • Add overlay[caller] before pragma[inline] on public, inline-marked predicates in dataflow modules to ensure they remain inlined across overlay frontiers.

@kaspersv kaspersv requested a review from aschackmull June 18, 2025 06:58
@aschackmull
Copy link
Contributor

  • skip library files that end with Query.qll or Config.qll if they contain implements DataFlow::ConfigSig

This is currently a fairly incomplete heuristic (there are other places that invoke global data flow), but I guess this provides a decent enough starting point and that further manual tweaking is to be expected.

@kaspersv
Copy link
Contributor Author

This is currently a fairly incomplete heuristic (there are other places that invoke global data flow), but I guess this provides a decent enough starting point and that further manual tweaking is to be expected.

Agreed. I think we should keep the current heuristic for now as we have tested the results extensively. However, if you have concrete ideas for extending the heuristic, I would be happy to try to incorporate them in the future.

@kaspersv
Copy link
Contributor Author

@hvitved @tausbn Would you mind reviewing the C# and Python parts of this PR (commit 2)? The PR adds overlay annotations for Java, but since we still have a few files that are sync'ed between languages, this also affects a few C# and Python files. Overlay compilation is still disabled for C# and Python and will remain so for the foreseeable future, so the annotations won't have any affect on compilation for C# or Python, beyond additional error checks. The overlay annotations are documented here and I would also be happy to give a quick intro to the overlay annotations as context.

@kaspersv kaspersv requested review from tausbn and hvitved June 20, 2025 05:30
@aschackmull
Copy link
Contributor

@hvitved @tausbn Would you mind reviewing the C# and Python parts of this PR (commit 2)?

You can take my approval as approval for C# and Python as well. 🦭

@kaspersv kaspersv removed request for tausbn and hvitved June 20, 2025 08:24
@kaspersv kaspersv force-pushed the kaspersv/overlay-annotations-script branch from 5f5e0d9 to ed8b787 Compare June 20, 2025 11:58
@kaspersv kaspersv force-pushed the kaspersv/overlay-java-annotations branch from ea40677 to 052023e Compare June 20, 2025 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants